Image acquisition method and apparatus

ABSTRACT

An image acquisition method operates in a hand held image acquisition device with a camera. A first image of a scene is obtained with the camera at a nominal exposure level. A number of relatively bright pixels and a number of relatively dark pixels within the first image are determined. Based on the number of relatively bright pixels, a negative exposure adjustment is determined and based on the number of relatively dark pixels, a positive exposure adjustment is determined. Respective images are acquired at the nominal exposure level; with the negative exposure adjustment; and with the positive exposure adjustment as component images for high dynamic range (HDR) image of the scene.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. Non-Provisional Patent Applicationclaiming the benefit of priority of U.S. Provisional Patent ApplicationNo. 62/147,464, filed on Apr. 14, 2015, the content of which isexpressly incorporated by reference herein in its entirety.

FIELD

The present invention relates to an image acquisition method andapparatus.

BACKGROUND

High-dynamic-range (HDR) photographs are typically generated byacquiring multiple component images of a scene, each with differentexposure levels, and then later, merging the component images into asingle HDR image.

Some prior art approaches can involve acquiring a set of, for example, 8component images, across a range of different exposure levels to ensurethat appropriately exposed image information is available for allregions of a HDR image. However, this can involve significant delay inacquiring the component images and so can result in blurring or ghostingartefacts from one image to the next. It can also require a large amountof processing to handle the set of component images.

U.S. Pat. No. 8,724,921, Aptina, discloses a method for capturing a highdynamic range (HDR) image. Multiple component images of a scene arecaptured at respectively different exposure settings. A further image ofan object placed in the scene is captured at one exposure setting. Afirst radiance image is formed from the multiple component images. Asecond radiance image is formed from the further image. The firstradiance image and the second radiance image are merged to form the HDRimage.

Natalia Gurieva “Complete Digital Workflow for HDR Photography”,International Circular of Graphic Education and Research, No. 7, 2014,pp 14-23 discloses capturing a HDR image including conducting a dynamicrange evaluation of the scene. Depending on the type of the scene andits dynamic range, different capturing strategies can be applied. Forexample, in case of midday sun with strong shadows it will be enough totake 3 shots at about 1 to about 1.33 stops apart; inside buildings withsome light coming through the windows at least 5 bracketed shots atabout 2 stops apart have to be taken.

U.S. Pat. No. 8,687,087, CSR Tech discloses a different approach tocapturing an image of a scene which would require high dynamic rangeincluding adjusting exposure time on an image block by block basis tomaintain a resulting digital signal within a range carried by a digitalprocessing path that carries a limited number of bits.

US 2014/0002694, CSR Tech discloses capturing two or more image framesusing different exposure settings and then combining the images to forma single HDR output frame in a video sequence. A pipelined architectureoperates on adjacent image frames by performing image alignment, imagemixing and tone mapping on the adjacent image frames to generate the HDRimage sequence.

It is an object of the present invention to provide an efficienttechnique for determining appropriate exposure levels for a limitednumber of component images used to provide a HDR image.

SUMMARY

According to a first aspect of the present invention there is providedan image acquisition method disclosed herein.

In still further aspects there is provided an image acquisition devicearranged to perform the methods disclosed herein.

Separately, there is provided a non-transient computer readable mediumcomprising computer executable instructions, which instructions whenexecuted on an image acquisition device, cause the image acquisitiondevice to perform the methods disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention will now be described, by way ofexample, with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of an image acquisition device on whichembodiments of the present invention can be implemented;

FIG. 2 is a flow diagram illustrating an image acquisition methodoperable on a device such as shown in FIG. 1;

FIG. 3(a) illustrate thresholding curves employed in the method of FIG.2;

FIG. 3(b) shows exemplary image data combined with the thresholdingcurves of FIG. 3(b); and

FIG. 3(c) illustrates exposure value conversion curves employed in themethod of FIG. 2.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to FIG. 1 which is a block diagram of an image acquisitiondevice 20, which in the present embodiment is a portable digital camera,operating in accordance with certain embodiments. It will be appreciatedthat many of the processes implemented in the digital camera areimplemented in or controlled by software operating on a microprocessor,central processing unit, controller, digital signal processor and/or anapplication specific integrated circuit, collectively depicted asprocessor 120. All user interface and control of peripheral componentssuch as buttons and display is controlled by a microcontroller 122.

In operation, the processor 120, in response to a user input at 122,such as half pressing a shutter button (pre-capture mode 32), initiatesand controls the digital photographic process. Ambient light exposurecan be determined using a light sensor 40 in order to automaticallydetermine an exposure setting for image acquisition. The distance to thesubject is determined using a focusing mechanism 50 which also focusesthe image on an image capture device 60. In a flash mode of the device,the processor 120 can cause a flash device 70 to generate a photographicflash in substantial coincidence with the recording of the image by theimage capture device 60 upon full depression of the shutter button.Flash mode may be selectively generated either in response to the lightsensor 40 or a manual input from the user of the camera.

The image capture device 60 digitally records acquired images in colour.The image capture device is known to those familiar with the art and mayinclude a CCD (charge coupled device) or CMOS to facilitate digitalrecording. High resolution images recorded by image capture device 60 isstored in an image store 80 which may comprise computer memory such adynamic random access memory or a non-volatile memory. The camera isequipped with a display 100, such as an LCD, both for displaying previewimages and displaying a user interface for camera control software.

In the case of preview images which are generated in the pre-capturemode 32 with the shutter button half-pressed, the display 100 can assistthe user in composing the image, as well as being used to determinefocusing and exposure. Temporary storage 82 is used to store one orplurality of the stream of preview images and can be part of the imagestore 80 or a separate component. The preview image is usually generatedby the image capture device 60. For speed and memory efficiency reasons,preview images may have a lower pixel resolution than the main imagetaken when the shutter button is fully depressed, and can be generatedby sub-sampling a raw captured image using software 124 which can bepart of the general processor 120 or dedicated hardware or combinationthereof.

In the present embodiment, a face detection and tracking module 130 suchas described for example, PCT Publication No. WO2008/018887 (Ref:FN-143), is operably connected to the sub-sampler 124 to control thesub-sampled resolution of the preview images in accordance with therequirements of the face detection and tracking module. Preview imagesstored in temporary storage 82 are available to the module 130 whichrecords the locations of faces tracked and detected in the preview imagestream. In one embodiment, the module 130 is operably connected to thedisplay 100 so that boundaries of detected and tracked face regions canbe superimposed on the display around the faces during preview.

In FIG. 1, the face tracking module 130 is arranged to extract and storetracked facial regions at relatively low resolution in a memory buffersuch as memory 82 and possibly for storage as meta-data in an acquiredimage header stored in memory 80. Where multiple face regions aretracked, a buffer is established for each tracked face region. Thesebuffers are of finite size (10-20 extracted face regions in a preferredembodiment) and generally operate on a first-in-first-out (FIFO) basis.

The device 20 can further include an image correction module 90 whichmay perform post processing of any acquired images. Where the module 90is arranged for off-line correction of acquired images in an externalprocessing device 10, such as a desktop computer, a colour printer or aphoto kiosk, face regions detected and/or tracked in preview images arepreferably stored as meta-data within the image header. However, wherethe module 90 is implemented within the camera 20, it can have directaccess to the buffer 82 where preview images and/or face regioninformation is stored.

The module 90 can thus receive the captured high resolution digitalimages from the store 80 and analyze these to improve the quality of theimage. The module can modify the image and the modified image may beeither displayed on image display 100, saved on a persistent storage 112which can be internal or a removable storage such as CF card, SD card orthe like, or downloaded to another device via image output means 110which can be tethered or wireless. The module 90 can be brought intooperation either automatically each time an image is captured, or uponuser demand via input 30. Although illustrated as a separate item, wherethe module 90 is part of the camera, it may be implemented by suitablesoftware on the processor 120.

In embodiments of the present invention, the image acquisition deviceprovides a HDR mode 72 selectable by the user of the image acquisitiondevice 20. Typically, HDR mode is chosen as an alternative to anon-flash automatic exposure mode; or a flash exposure mode; or indeedany number of other specialised image acquisition modes such as portraitor panorama modes.

Referring to FIG. 2, the mode is typically implemented by the processor120 as follows: An image of a scene is captured at a nominal exposurelevel (0EV), step 200. The 0EV exposure level for this image can bedetermined based on an average light level of a scene provided by thelight sensor 40; or can be determined based on spot metering of thescene; or can be determined based on a exposure level of for example,one or more face regions detected within the scene by the face tracker130.

The 0EV image can be a preview image and displayed on the image display100 as normal. Once the 0EV image is acquired, it is analysed in step202 to determine the exposure levels for the remaining component imagesto be used in producing a HDR image.

This analysis first of all comprises computing a histogram of pixelintensities for the 0EV image. It will be appreciated that thistechnique lends itself to processing images in formats where intensityis provided in a separate image plane. Nonetheless, images can beacquired in any number of formats including RGB, LAB, YCC, YUV etc. Insome cases such as LAB, YCC or YUV, one of the colour planes L or Yprovides an intensity value 0≤Intensity≤Maxlevel, typically 255 for 8bit pixels, directly, whereas for others such as RGB, colour planeinformation needs to be combined or transformed to provide an intensityvalue, for example, Intensity=0.2126*R+0.7152*G+0.0722*B.

Referring now to FIG. 3(b), a histogram for an image (histImg) acquiredfrom a typical scene is shown.

The analysis now uses respective dark and light thresholding (orweighting) curves (curveDark, curveLight), FIG. 3(a), each curve rangingin value from 0 to 1, to obtain a measure of the extent of dark andburned areas within the image.

In the example, the dark thresholding curve (curveDark) is zero valuedabove a relatively low corner intensity value (DT) and increases invalue towards 1 at zero intensity. Conversely, the light thresholdingcurve (curveLight) is zero valued below a relatively high cornerintensity value (LT) and increases in value towards 1 at maximumintensity (in this case 255). Each curve increases monotonically with afirst low inflection point and a second higher inflection point, withcurve values tending to plateau as they increase towards 1. As such,each of curveDark and curveLight can be thought of as low pass and highpass filters for image histogram information.

The thresholding curves are based on the observation that in a wellilluminated scene most pixel intensity values are the middle of therange. Some well illuminated scenes can still include dark and lightpixels, but these should be relatively low in frequency, and need notnecessarily cause a dynamic range problem.

If DT and LT alone were used as simple thresholding values fordetermining the number of dark and bright pixels in an image and toprovide indicators of the relative darkness or lightness of an image,then exposure adjustments for images including normally occurring darkand bright pixels would tend to be too great. The thresholding curves ofFIG. 3(a) therefore tend to limit the effect of most bright and mostdark pixels on the exposure adjustments which may be made for additionalcomponent images.

By multiplying the values for histImg against curveDark and curveLightas follows:darkPrc=W _(D)*Σ_(i=0) ²⁵⁵(histImg_(i)*curveDark_(i))lightPrc=W _(L)*Σ_(i=0) ²⁵⁵(histImg_(i)*curveLight_(i))respective measures of the extent of dark areas within the image(indicated as Dark Areas in FIG. 3(b)) and light areas within the image(indicated as Burned Areas in FIG. 3(b)) can be provided. In theexample, a weight W_(D)=W_(L)=100 is used to scale the summed values andcan be dependent on the profile of the thresholding curves. For exampleW_(D)*∫curveDark=W_(L)*∫curveLight=1. In the example of FIG. 3(b), theregion indicated as “dark areas” illustrates the product of histImg andcurveDark; while the region indicated as “burned areas” illustrates theproduct of histImg and curveLight. The values for these regions aresummed (integrated) and scaled to provide darkPrc, lightPrc.

In the example, of FIG. 3(b), burned areas indicates a greater amount ofsaturated image area than dark areas.

These resultant values darkPrc and lightPrc are now used as lookupvalues to determine an initial plusEV and minusEV adjustment for twocomponent images to be subsequently acquired as follows:minusEV=−minusCurve(lightPrc);plusEV=plusCurve(darkPrc);

The values within the lookup table are determined experimentally andincrease in proportion to the darkPrc and lightPrc figures respectively.It will be noted that for a given area of dark pixels (darkPrc), theabsolute value of the overexposure adjustment level (plusEV) will beless than the absolute value of the underexposure adjustment level(minusEV) for the same area of bright pixels (lightPrc).

Referring back to FIG. 2, in the example, a minusEV of −0.7 and a plusEVof 0.5 are determined for an 0EV image. Exposure settings are adjustedaccordingly and a respective preview image can be captured at each ofthese settings 204 a, 204 b; or indeed a relatively short exposure time(SET) image can be acquired at the minusEV setting and exposure allowedto continue without clearing the sensor before acquiring a longerexposure time (LET) image at the plusEV setting.

As an alternative or in addition to adjusting exposure time, if theimage acquisition device 20 comprises an adjustable aperture, thenaperture stop can also be adjusted to adjust the exposure of the imagescaptured in steps 204 a and 204 b. However, in this case such imageswould have to be acquired successively.

Each of the SET and LET images are then analysed in the same manner asthe 0EV image in steps 202 a and 202 b, except in the case of the SETimage, only lightPrc is of concern; whereas for the LET image, onlydarkPrc is of concern.

If the analysis 202 a of the SET image provides lightPrc with anabsolute value >0, then the minusEV value is adjusted further, step 204aa.

Similarly, if the analysis 202 b of the SET image provides darkPrc withan absolute value >0, then the plusEV value is adjusted further, step204 bb.

In the example, the final minusEV and plusEV values are chosen as −1.2EVand +0.8EV respectively.

Now each of a final minusEV and plusEV image are acquired at steps 200 aand 200 b respectively. Again, the minusEV image can be acquired at theminusEV setting and exposure allowed to continue without clearing thesensor before acquiring the longer exposure time plusEV image at theplusEV setting. In this case, each of the minusEV, a second 0EV andplusEV images can be acquired successively from the sensor to mitigateproblems with blur or ghosting between the minusEV and plusEV images andthe 0EV image acquired at step 200.

Again, as an alternative or in addition to adjusting exposure time, ifthe image acquisition device 20 comprises an adjustable aperture, thenaperture stop can also be adjusted to adjust the exposure of the imagescaptured in steps 200 a and 200 b. Again, in this case such images wouldhave to be acquired successively.

In this case, where the 0EV image acquired at step 200 is not used as acomponent image in HDR processing, a full resolution image need not becaptured at step 200.

Where the minusEV and plusEV images are acquired separately, each of the0EV and these images may need to be aligned before they can be combinedinto a HDR image and there can be problems with motion blur where theseis movement within a scene.

On the other hand, where the minusEV and plusEV and possibly 0EV imagesare acquired substantially contemporaneously, ghosting artefacts canarise and special processing may be required to deal with theseartefacts.

In any case, once the 0EV, minusEV and plusEV component images areavailable, they can be combined at step 206 into a HDR image. Thisprocessing can be performed immediately so that the HDR image can beviewed immediately on the acquisition device 20, or for example, theprocessing can be performed with a post processing module such as thecorrection module 90.

It will be appreciated that where the relatively short exposure time andrelatively long exposure time images acquired in steps 204 a and 204 bare full resolution images, these could also be retained for use ascomponent images of a HDR image.

It will be appreciated that each of the analysis steps 202, 202 a and202 b need not analyse every pixel of an input image. So for example, ifa full-resolution preview image has been acquired, it can be sub-sampledfor the purposes of determining minusEV and plusEV. In any case, it willbe seen that the above technique enables an image acquisition device toquickly acquire a minimal set of HDR images at suitable exposuresettings for subsequent use in creating a HDR image.

The invention claimed is:
 1. A system comprising: one or moreprocessors; an image sensor; memory comprising computer executableinstructions that, when executed by the one or more processors, causethe system to perform operations comprising: obtaining a first image viathe image sensor based at least in part on a first exposure level;identifying a first set of pixels of the first image based at least inpart on a light thresholding curve and first intensities associated withthe first set of pixels; identifying a second set of pixels of the firstimage based at least in part on a dark threshold curve and secondintensities associated with the second set of pixels; determining anunderexposure adjustment level based at least in part on the first setof pixels; determining an overexposure adjustment level based at leastin part on the second set of pixels; obtaining, via the image sensor, asecond image based at least in part on the underexposure adjustmentlevel; and obtaining, via the image sensor, a third image based at leastin part on the overexposure adjustment level.
 2. The system of claim 1,wherein obtaining the first image comprises: obtaining an image via theimage sensor; and sub-sampling the image to obtain the first image. 3.The system of claim 1, wherein determining the underexposure adjustmentlevel comprises: obtaining a fourth image via the image sensor based atleast in part on the first exposure level and the underexposureadjustment level; sub-sampling the fourth image; and modifying, based atleast in part on the first set of pixels, the underexposure adjustmentlevel.
 4. The system of claim 1, wherein determining the overexposureadjustment level comprises: obtaining a fourth image via the imagesensor based at least in part on the first exposure level and theoverexposure adjustment level; sub-sampling the fourth image; andmodifying, based at least in part on the second set of pixels, theoverexposure adjustment level.
 5. The system of claim 1, wherein:determining the underexposure adjustment level comprises: filtering ahistogram associated with the first image based at least in part on thelight thresholding curve to obtain a first filtered histogram, andintegrating the first filtered histogram; determining the overexposureadjustment level comprises: filtering the histogram associated with thefirst image based at least in part on the dark thresholding curve toobtain a second filtered histogram, and integrating the second filteredhistogram; and the histogram comprises luminance values associated withpixels of the first image.
 6. The system of claim 1, wherein theoperations further comprise generating a high dynamic range (HDR) imagebased at least in part on the second image and the third image.
 7. Thesystem of claim 1, wherein the light thresholding curve is based atleast in part on a high pass filter and the dark thresholding curvecomprises is based at least in part on a low pass filter.
 8. The systemof claim 7, wherein: a first corner frequency of the low pass filter isless than twenty percent of a maximum intensity indicatable by anintensity value associated with a pixel; and a second corner frequencyof the high pass filter is more than seventy-eight percent of themaximum intensity.
 9. A method comprising: obtaining a first image viaan image sensor based at least in part on a first exposure level;identifying a first set of pixels of the first image based at least inpart on a high pass filter and first intensities associated with thefirst set of pixels; identifying a second set of pixels of the firstimage based at least in part on a low pass filter and second intensitiesassociated with the second set of pixels; determining an underexposureadjustment level based at least in part on the first set of pixels;determining an overexposure adjustment level based at least in part onthe second set of pixels; obtaining, via the image sensor, a secondimage based at least in part on the underexposure adjustment level; andobtaining, via the image sensor, a third image based at least in part onthe overexposure adjustment level.
 10. The method of claim 9, whereindetermining the underexposure adjustment level comprises: obtaining afourth image via the image sensor based at least in part on the firstexposure level and the underexposure adjustment level; sub-sampling thefourth image; and modifying, based at least in part on the first set ofpixels, the underexposure adjustment level.
 11. The method of claim 9,wherein determining the overexposure adjustment level comprises:obtaining a fourth image via the image sensor based at least in part onthe first exposure level and the overexposure adjustment level;sub-sampling the fourth image; and modifying, based at least in part onthe second set of pixels, the overexposure adjustment level.
 12. Themethod of claim 9, wherein: determining the underexposure adjustmentlevel comprises: filtering a histogram associated with the first imagebased at least in part on the light thresholding curve to obtain a firstfiltered histogram, and integrating the first filtered histogram;determining the overexposure adjustment level comprises: filtering thehistogram associated with the first image based at least in part on thedark thresholding curve to obtain a second filtered histogram, andintegrating the second filtered histogram; and the histogram comprisesluminance values associated with pixels of the first image.
 13. Themethod of claim 9, further comprising generating a high dynamic range(HDR) image based at least in part on the second image and the thirdimage.
 14. The method of claim 9, wherein: a first corner frequency ofthe low pass filter is less than twenty percent of a maximum intensityindicatable by an intensity value associated with a pixel; and a secondcorner frequency of the high pass filter is more than seventy-eightpercent of the maximum intensity.
 15. A non-transitory computer-readablemedium comprising processor-executable instructions that, when executedby one or more processors, cause the one or more processors to performoperations comprising: obtaining a first image via an image sensor basedat least in part on a first exposure level; identifying a first set ofpixels of the first image based at least in part on a high pass filterand first intensities associated with the first set of pixels;identifying a second set of pixels of the first image based at least inpart on a low pass filter and second intensities associated with thesecond set of pixels; determining an underexposure adjustment levelbased at least in part on the first set of pixels; determining anoverexposure adjustment level based at least in part on the second setof pixels; obtaining, via the image sensor, a second image based atleast in part on the underexposure adjustment level; and obtaining, viathe image sensor, a third image based at least in part on theoverexposure adjustment level.
 16. The non-transitory computer-readablemedium of claim 15, wherein determining the underexposure adjustmentlevel comprises: obtaining a fourth image via the image sensor based atleast in part on the first exposure level and the underexposureadjustment level; sub-sampling the fourth image; and modifying, based atleast in part on the first set of pixels, the underexposure adjustmentlevel.
 17. The non-transitory computer-readable medium of claim 15,wherein determining the overexposure adjustment level comprises:obtaining a fourth image via the image sensor based at least in part onthe first exposure level and the overexposure adjustment level;sub-sampling the fourth image; and modifying, based at least in part onthe second set of pixels, the overexposure adjustment level.
 18. Thenon-transitory computer-readable medium of claim 15, wherein:determining the underexposure adjustment level comprises: filtering ahistogram associated with the first image based at least in part on thelight thresholding curve to obtain a first filtered histogram, andintegrating the first filtered histogram; determining the overexposureadjustment level comprises: filtering the histogram associated with thefirst image based at least in part on the dark thresholding curve toobtain a second filtered histogram, and integrating the second filteredhistogram; and the histogram comprises luminance values associated withpixels of the first image.
 19. The non-transitory computer-readablemedium of claim 15, further comprising generating a high dynamic range(HDR) image based at least in part on the second image and the thirdimage.
 20. The non-transitory computer-readable medium of claim 15,wherein: a first corner frequency of the low pass filter is less thantwenty percent of a maximum intensity indicatable by an intensity valueassociated with a pixel; and a second corner frequency of the high passfilter is more than seventy-eight percent of the maximum intensity.